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1  Introduction 

This  technical  report  summarizes  our  findings  obtained  under  the  aegis  of  the  AFOSR  grant 
in  a  variety  of  areas  related  to  computation  and  learning  with  particular  reference  to  neural 
network  settings.  The  main  results  obtained  have  been  in  problems  related  to:  analyzing 
and  characterizing  random  graph  phenomena  in  algorithms  learning  binary  weights  for  neu¬ 
rons  [l]-[7];  optimal  stopping  and  effective  machine  complexity  in  network  learning  [18]- 
[28];  nonparametric  learning,  with  specific  reference  to  the  finite  sample  behavior  of  the 
k-nearest  neighbor  classifier  [9],  [12]-[15];  learning  from  mixtures  of  labeled  and  unlabeled 
examples  [10]  [11];  and  issues  in  enumeration,  computation,  and  learning  [8]  [12],  [16],  [17]. 
We  summarize  our  main  findings  in  each  of  these  areas  eschewing  technical  detail  and  formal 
proofs  of  the  various  assertions.  These  may  be  found  in  the  various  papers  referenced  above 
which  are  listed  at  the  end  of  the  report. 


2  Learning  Binary  Weights  and  Random  Graphs 

Consider  the  following  classical  problem  in  mathematical  programming.  Write  B  =  {— 1 , 1} 
for  simplicity,  and  let  =  {— 1 denote  the  vertices  of  the  n-cube.  Let  U  =  < 

oc  <  m)  be  an  m-set  of  vertices  (patterns,  examples,  or  simply,  points)  in  B^  and  let 
£  =  <a<m}bea  corresponding  set  of  signs  (classifications,  labels,  or  colors), 

P  e  B,  which  two-colors  the  m-set  of  vertices  U  into  “positive”  and  “negative”  examples. 
We  are  interested  in  the  following  question:  Can  we  find  a  hyperplane  separating  the  positive 
examples  firom  the  negative  examples? 

More  formally,  for  x  =  (xi , . . .  ,Xn)  and  y  =  (pi , ...  ,'yn),  let  us  denote  the  standard 
inner  product  in  by  (x,y)  ==  XiPi.  We  seek  to  find  a  vector  of  weights  w  = 

(wi , . . . ,  Wn)  such  that,  for  each  example  =  (u^, . . .  ,u^)  €  U, 


(w,u“)=X^ 


i=1 


< 

> 


if  1“  =-1, 
ifl“  =  +l. 


(1) 
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If  there  exists  such  a  w,  then  the  linear  subspace  (hyperplane)  orthogonal  to  w  will  separate 
the  positive  examples  from  the  negative  examples. 

Before  we  proceed  further,  let  us  get  rid  of  a  minor  notational  nuisance  by  agreeing  to 

reflect  all  negative  example  about  the  origin.  In  particular,  if  =  —  1,  set  i - and 

i — hi .  An  examination  of  the  first  inequality  in  (1)  shows  that  this  is  perfectly  legitimate 
as  (w,u°^)  <  0  iff  (w,— >  0.  Thus,  without  loss  of  generality,  we  can  assume  that  the 
TTL-set  of  vertices  U  consists  of  only  positive  examples.  Our  programming  problem  then  is 
to  determine  a  weight  vector  w  such  that 

(w,u)>0  (ueW),  (2) 

if  any  such  w  exists.  The  two  formulations  (1)  and  (2)  are  equivalent.  In  the  sequel 
it  will  be  found  natural  to  set  up  probability  distributions  on  labeled  examples  (u,l)  for 
the  framework  (1);  notational  convenience  will  dictate,  however,  that  we  proceed  with  the 
analysis  using  the  induced  probability  distributions  for  the  equivalent  framework  (2)  wherein 
all  negative  examples  are  reflected  around  the  origin. 

If  w  is  allowed  to  range  over  Euclidean  n-space  R^,  the  corresponding  decision  problem 
is  simple:  the  problem  is  just  an  instance  of  LINEAR  PROGRAMMING  and  has  a  poly¬ 
nomial  time  solution,  for  instance,  using  interior  point  methods,^  Note  that  in  this  case, 
the  transformed  problem  (2)  can  be  posed  in  an  equivalent  geometric  formulation:  Does 
the  convex  hull  of  the  m-set  of  vertices  U  contain  the  origin?  If  the  answer  is  affirmative 
then  (2)  has  no  solution;  if  the  answer  is  negative  then  a  solution  w  to  (2)  exists. 

The  problem  becomes  much  harder  if  putative  solutions  w  are  allowed  to  range  only 
over  the  vertices  of  the  cube.  The  corresponding  decision  problem  in  this  case  is  an 
instance  of  BINARY  INTEGER  PROGRAMMING  which  is  known  to  be  NP-complete. 
Our  focus  will  be  on  algorithmic  approaches  to  this  problem.  Before  we  proceed,  let  us 
recast  the  problem  as  a  learning  problem  in  neural  networks  in  a  form  which  will  allow  of  a 
substantive  generalization. 


2.1  The  Binary  Percept ron 

A  formal  neuron  in  the  model  of  McCulloch  and  Pitts  is  a  linear  threshold  element  which 
produces  as  output  the  sign  of  a  linear  form  of  its  inputs.  Viewed  as  a  logic  element,  a  neuron 
characterized  by  real  weights  w  =  (wi , . . . ,  Wn)  maps  Boolean^  inputs  u  =  (ui , . . .  ,Urt)  € 
B^  into  Boolean  outputs  v  e  B  in  accordance  with  the  threshold  rule: 


V  =  sgn 


sgn(w,u) 


— 1’  if{w,u)<0, 
-hi  if(w,u)>0. 


(3) 


It  is  not  hard  to  see  that  neurons  form  a  universal  basis  for  Boolean  functions;  for  instance, 
a  two-input  NAND  is  computed  by  a  two-input  neuron  whose  weights  are  identically  —1 , 
thus:^ 

NAND(ui,U2)  =sgn(“Ui  —  U2). 


^Bibliographical  notes  and  references  to  original  work  may  be  found  in  the  concluding  section. 

^It  is  simpler  in  this  framework  to  adopt  the  convention  that  truth  values  are  represented  by  +1  (for 
true)  and  —1  (for  false)  instead  of  the  usual  1  and  0. 

^  We’ve  explicitly  made  use  of  the  definition  sgn  0=1.  If  this  expedient  definition  is  too  much  for  the 
gentle  reader  to  swallow,  she  may  replace  the  two-input  neuron  by  a  three-input  neuron,  two  of  whose  inputs 
are  literals,  with  the  third  input  a  constant  —1 .  With  the  selection  of  all  weights  —1  again,  it  is  simple 
to  see  that  NAND(ui  , U2)  =  sgn(— ui  —  ui  +  1 ). There  are  advantages  to  considering  this  more  robust 
representation  in  the  presence  of  noise. 
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In  particular,  for  any  Boolean  function  there  exists  a  network  of  formal  neurons  which 
computes  it. 

The  power  of  the  neural  network  computational  paradigm  rests  in  part  on  the  fact  that 
the  basic  linear  threshold  computational  element  is  substantially  more  powerful,  viewed  as 
a  logic  element,  than  standard  logic  elements.  Indeed,  a  classical  result  asserts  that  of  the 
2^""  possible  Boolean  functions  of  n  Boolean  variables,  a  formal  neuron  can  compute 
Boolean  functions  by  varying  the  weights.  This  investiture  of  substantial  computational 
power  in  each  element  potentially  results  in  significant  improvements  in  circuit  complexity. 

The  second  leg  of  the  appeal  of  networks  of  formal  neurons  (and  sigmoidal  generaliza¬ 
tions  where  the  sign  operation  is  replaced  by  a  sigmoidal  function)  as  a  universal  computa¬ 
tional  paradigm  rests  on  the  availability  of  simple  learning  algorithms  such  as  the  Perceptron 
training  algorithm  (for  a  single  neuron)  and  Backpropagation  (for  sigmoidal  networks)  which 
the  circuit  designer  can  utilize  to  prescribe  the  weights  of  the  network. 

While  most  uses  of  neural  networks  involve  selections  of  real  weights  for  the  network,  it 
is  clear  from  the  NAND  example  that  binary  weights  suffice.  Let  us  call  the  computational 
device  represented  by  the  input-output  relation  (3)  a  binary  neuron  or  binary  perceptron  if 
the  weights  are  constrained  to  take  values  in  B  only.  It  is  then  clear  that  a  binary  neuron 
forms  a  universal  basis  for  Boolean  functions.  Thus,  any  network  of  neurons  with  real 
weights  can  be  replaced  by  an  equivalent,  albeit  larger,  network  of  neurons  with  binary 
weights.  While  implementational  cost  considerations  favor  networks  with  binary  weights, 
the  practical  utility  of  networks  of  binary  neurons  will  devolve  in  significant  measure  upon 
whether  there  exist  efficient  algorithms  for  specifying  binary  weights  for  a  network.  The 
general  learning  problem  here  can  be  stated  as  follows:  Given  a  labeled  set  of  vertices  and  a 
network  architecture,'^  can  we  learn  binary  weights  for  the  network  which  yield  a  consistent 
classification  of  the  set  of  examples?  In  particular,  the  particular  instance  of  the  BINARY 
INTEGER  PROGRAMMING  problem  embodied  in  (2)  is  a  special  case  of  the  general 
problem:  can  we  learn  binary  weights  for  a  (binary)  perceptron  which  yields  a  consistent 
classification  of  a  given  set  of  labeled  vertices? 

While  the  worst-case  intractability  of  the  particular  problem  of  learning  binary  weights 
for  a  neuron  also  implies  the  worst-case  intractability  of  the  general  problem  of  learning  bi¬ 
nary  weights  for  networks,  nevertheless,  as  we  see  in  the  sequel,  we  can  design  (randomized) 
algorithms  which  very  efficiently  learn  binary  weights  consistently  for  almost  all  instances 
of  the  problem.  We  focus  here  on  three  recent  algorithmic  approaches  that  we  have  pro¬ 
posed  and  analyzed  to  learning  binary  weights  for  perceptrons.  While  the  results  will  be 
specialized  to  the  BINARY  INTEGER  PROGRAMMING  problem,  the  principles  behind 
the  algorithms  can  be  readily  extended  to  the  general  problem  of  learning  binary  weights 
for  fixed  architectures  of  binary  neurons. 

The  analysis  of  the  peccadilloes  of  individual  algorithms  waits  upon  the  specification  of 
the  probability  space  in  which  the  labeled  examples  reside,  and  two  distinct  scenarios  emerge 
depending  on  the  mechanism  by  which  the  labeled  examples  are  specified.  The  probability 
spaces  will  be  found  to  arise  most  naturally  in  the  setting  of  the  original  problem  (1)  so  let 
us  refer  back  to  (1)  for  the  nonce  to  set  up  the  two  statistical  scenarios  of  interest;  notational 
expedience  will  dictate,  however,  that  we  subsequently  work  with  the  induced  distributions 
for  the  equivalent  Problem  (2)  obtained  by  reflecting  negative  examples  about  the  origin. 

We  assume  in  both  scenarios  that  {  :  1  <  cx  <  m},  the  set  of  labeled  examples 

corresponding  to  Problem  (1),  is  drawn  by  independent  sampling  from  a  given  probability 
distribution  on  xB.  Furthermore,  we  assume,  again  for  both  scenarios,  that  the  marginal 

network  architecture  is  specified  by  an  interconnectivity  graph  on  a  set  of  neurons  (which  comprises 
the  vertices  of  the  graph)  together  with  the  identification  of  input  neurons  and  the  output  neuron. 
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distribution  of  the  examples  is  uniform  over  the  vertices  of  the  cube.  Equivalently, 
the  pattern  components  {uf,l  <i<Ti,  1  <a<Ta}  form  an  independent,  identically 
distributed  sequence  of  symmetric  Bernoulli  random  variables  satisfying 

P{uf  =  --1}  =  P{uf  =  +1}  =:  1/2. 

It  suffices  now  to  specify  the  conditional  distribution  of  the  labels  1“  given  the  examples 
VL^.  Two  cases  arise  naturally:  in  the  first,  the  labels  1°^  are  specified  independently  of  the 
corresponding  examples  from  an  arbitrary  probability  distribution  on  B;  in  the  second, 
given  the  examples  u^,  the  labels  P  are  specified  in  a  completely  deterministic  fashion 
according  to  the  labeling  of  the  examples  by  a  fixed  target  perceptron  G  B^.  We 
consider  these  two  cases  in  turn. 

Random  Problems  In  our  context,  we  say  that  a  binary  integer  programming  Prob¬ 
lem  (1)  is  random  if  examples  and  their  corresponding  labels  are  specified  independently  of 
one  another.  In  particular,  the  set  of  labels  C  =  {P}  forms  an  i.i.d.  sequence  of  random 
variables,  independent  of  the  set  of  examples  and  with  a  common  (not  necessarily 
symmetric)  Bernoulli  distribution  on  B. 

While  it  is  clear  that  for  any  m-set  of  points  U  there  exists  at  least  one  labeling  C 
which  is  linearly  separable  (in  the  sense  of  (1)),  it  is  not  difficult  to  see  that  the  converse 
is  also  true  for  large  enough  m.  Indeed,  when  m  >  n  there  necessarily  exists  at  least  one 
labeling  which  cannot  be  linearly  separated  by  a  binary  weight  vector.  A  simple  counting 
argument  establishes  this:  there  are  2^^  possible  labelings  of  an  m-set  of  points,  and  only  2^ 
distinct  binary  weight  vectors.  It  is  clear  further  that  the  probability  that  any  independently 
specified  coloring  of  the  set  of  patterns  is  linearly  separable  becomes  arbitrarily  small  when 
m  becomes  suitably  large  compared  to  n,  i.e.,  with  high  probability  there  will  exist  no 
solution  vector  for  (1).  Consequently,  any  binary  learning  algorithm  will  ultimately  falter 
when  the  set  of  random  examples  becomes  large  enough.  (Of  course,  some  algorithms 
may  fail  somewhat  earlier,  even  when  solutions  exist,  because  of  a  deficiency  inherent  in  the 
algorithm  itself.)  We  can  hence  pose  the  following  question  of  interest:  What  is  the  ^^largest” 
sample  size  m  for  which  a  given  algorithm  mil,  with  high  probability,  yield  a  solution  vector? 
Suitably  formalized  as  an  asymptotic  threshold  notion  this  measure  of  the  capacity  function 
of  an  algorithm  is,  loosely  speaking,  the  “largest”  size  m  =  of  random  binary  integer 
programming  problem  that  can  be  accommodated  by  the  algorithm. 

Linearly  Separable  Problems  At  the  other  end  of  the  spectrum,  labels  for  randomly 
drawn  examples  for  Problem  (1)  are  specified  in  a  fully-determined,  example-dependent 
fashion  according  to  an  underlying  target  perceptron  (or  concept).  In  particular,  let  w®  G  B^ 
be  some  fixed  (but  unknown)  vertex.  For  a  =  1 ,  . . . ,  m,  set 

^+1  if  (w®,u“}  >  0. 

Thus,  for  any  sample  size  m  we  are  always  guaranteed  the  existence  of  at  least  one  solution 
vector  satisfying  (1),  namely  w®,  and  an  exhaustive  search  algorithm,  for  instance,  will 
always  yield  a  consistent  hypothesis  whatever  the  sample  size  m.  The  previous  notion  of 
capacity  (or  “largest  size”  of  random  problem  that  an  algorithm  can  accommodate)  hence 
does  not  have  a  particularly  interesting  counterpart  in  this  linearly  separable  setting.  Intu¬ 
ition  suggests,  however,  that  for  a  large  enough  sample  size  m,  a  solution  vector  satisfying 
the  loading  problem  (1)  will  also  be  a  good  approximation  to  the  underlying  concept  w® 
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labeling  the  examples.  We  can  hence  pose  the  following  question  of  interest:  What  is  the 
^^smallest”  sample  size  m  for  which  a  given  algorithm  will,  with  high  probability,  identify 
the  unknown  target  perceptron  w®  as  the  unique  solution  vector  satisfying  (1)?  Suitably 
formalized  as  an  asymptotic  threshold  notion,  we  call  this  “smallest”  sample  size  m  =  mn 
the  sample  complexity  function  of  the  algorithm. 

This  is  a  problem  familiar  from  learning  theory.  The  output  of  the  binary  perceptron 
w  G  (where  weVe  tacitly  adopted  the  notational  convenience  of  identifying  a  perceptron 
by  its  weight  vector)  is  the  Majority  function 

hw(u)  =sgn(w,u), 

and  the  functions  to  be  learned  from  examples  (the  hypotheses)  are  exactly  the  set  of 
functions  {Hw  :  w  €  B^ ),  i.e.,  Majority  functions  of  a  set  of  literals.  We  may  identify  each 
hypothesis  function  Iv  simply  by  the  corresponding  vertex  w  6  B^.  In  this  setting,  the 
goal  of  learning  is  to  produce  a  hypothesis  w  (as  a  function  of  the  set  of  examples)  which 
approximates  the  underlying  target  concept  w®  which  labels  the  examples. 

In  the  sequel,  we  will  place  somewhat  more  stringent  requirements  on  the  learning  pro¬ 
cess:  we  will  require  perfect  learning  (or  perfect  generalization)  wherein  the  target  concept 
w®  labeling  the  examples  is  exactly  identified  (with  high  probability)  by  the  algorithm.  The 
“smallest”  sample  size  m  for  which  the  algorithm  can  achieve  this  identification  with  high 
probability  is  the  “complexity”  of  sample  demanded  by  the  algorithm  to  achieve  perfect 
generalization. 

2.2  Harmonic  Update 

Consider  the  following  on-line  situation.  The  teacher  generates  a  sequence  of  m  positive 
examples  u\  ...  ,  and  presents  them  in  turn  to  the  learner.  Each  example  is  presented 
only  once,  and  in  particular,  the  training  sequence  (u[t]}  consists  of  the  examples  u\  , . .  , 
in  succession:  u[t]  =  u^,  1  <  t  <  m.  The  learning  sequence  {w[t]}  of  weight  vector  updates 
consequently  terminates  in  m  steps  with  w[l]  =  (wi  [1], . . .  ,Wn[l])  €  B^  an  arbitrary  initial 
weight  vector  and  w  =  w[Tn  + 1]  the  final  weight  vector  returned  by  the  algorithm.  The  lack 
of  memory  poses  an  immediate  problem  here:  at  each  epoch  t,  the  learner  has  access  only 
to  her  current  estimate  of  the  weight  vector  w[t]  G  B’^,  the  current  example  u[t]  =  G  B^, 
and  the  current  epoch  t  G  {1 , . . . ,  m},  and  as  each  example  is  seen  only  once,  memory  cannot 
be  built  up,  as  in  other  on-line  scenarios,  by  repeated  presentations  of  the  examples. 

Harmonic  Update  is  an  on-line,  homogeneous  algorithm  which  dynamically  incorporates 
long-term  memory  of  each  example — single  presentation  notwithstanding — in  the  (binary) 
weights  by  using  auxiliary  randomization  to  retain,  at  each  epoch  t,  and  for  each  i  G 
a  maximal  amount  of  information  about  each  of  the  pattern  components  u®, 
1  <  s  <  t  in  the  corresponding  weight  Wi[t].  The  weight  updates  are  iteratively  specified  as 
follows: 

Algorithm  H  {Harmonic  Update).  Given  examples  u[t]  =  =  (u^ , . . .  ,u^)  G  B^  pre¬ 

sented  sequentially  at  epochs  t  =  1 ,  . . .  ,  m,  the  algorithm  recursively  generates  a  sequence 
of  binary  weight  vectors  w[t  -f  1]  =  (wi  [t  +  1], . . . ,  Wn[t  +  1])  G  B^  where  w[t  +  1]  is  a 
random  function  of  w[t]  and  u[t]  only.  After  m  epochs,  the  algorithm  returns  the  final 
weight  vector  w  =  w[m+  1]  as  a  putative  vertex  solution  to  the  system  of  inequalities  (2). 

HI.  [Initialize.]  Set  w[1]  =  (wi  [1], . , .  .Wnil])  to  be  an  arbitrary  vertex  in  B^.  Set  t  1 . 

H2.  [New  example.].  Obtain  example  u^. 
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H3.  [Reinitialize  component  index.]  Set  i  f-  1 . 

H4.  [Update  memory  components.]  If  Wi[t]  =  u^,  set  >Vi[t -f  1]  =  Wi[t];  else  if  Wi[t]  = 
set 

.  [t  +  1  ]  -  I  probability  1  /t, 

1  +Wi[t]  with  probability  1  —  1/t. 


H5.  [Iterate.]  Set  i  <—  i  -f  1 .  If  i  <  n,  go  back  to  step  H4;  otherwise  set  t  t  +  1 .  If  t  <  m 

go  back  to  step  H2;  otherwise  set  w  =  w[m+  1]  and  terminate  the  algorithm. 

The  intuition  behind  this  randomized  procedure  is  to  try  to  obtain  as  large  a  covariance 
between  each  weight  Wi  and  each  of  the  corresponding  pattern  components  as  possible. 
If  this  is  achieved,  the  summands  in  the  sum  WiUf  of  (2)  are  more  likely  to  be  positive, 
so  that  the  chances  of  the  entire  sum  being  positive  are  improved.  Now,  at  any  epoch  t, 
weight  Wilt]  contains  information  about  the  ith  components  of  the  first  t  —  1  patterns.  If 
the  ith  component  of  example  u*  has  the  same  sign  as  Wi[t]  there  is  no  problem.  The 
difficulty  arises  if  Wi[t]  and  have  opposite  signs:  not  changing  the  sign  of  Wi[t]  will  then 
lose  information  about  pattern  u^,  while  changing  the  sign  of  Wi[t]  will  result  in  a  loss 
of  information  about  the  first  t  —  1  patterns.  The  solution  in  this  case  is  to  change  the 
sign  of  the  weight  probabilistically,  and  with  increasing  reluctance  as  time  passes  (when 
there  is  presumably  considerable  past  history  stored  in  the  weight).  The  exact  measure  of 
this  reluctance  to  change  the  sign  of  the  weight  is  given  probabilistically  by  the  harmonic 
sequence  1  /t.  The  effect  of  this  randomized  procedure  is  to  ensure  that  each  weight  retains 
an  equal  amount  of  information  about  the  corresponding  component  of  every  pattern.  In 
particular,  we  have  uniformly  large  covariances  EWiU^  =  0(m“^). 

The  following  is  a  capsule  summary  of  the  main  features  of  the  algorithm.  Formal 
proofs  and  technical  details  may  be  found  in  the  papers  listed  at  the  end  of  the  report. 

•  Harmonic  Update  algorithm  is  on-line,  homogeneous,  and  randomized. 

•  The  algorithm  has  particularly  parsimonious  space  and  time  requirements:  it  requires 
a  buffer  memory  of  2n  +  [log  m]  bits  and  a  time  complexity  of  ra  epochs  (or  a  serial 
time  complexity  of  mn  bit  steps). 

•  Notwithstanding  its  low  complexity  of  specification,  the  algorithm  has  a  surprisingly 
large  capacity  ^/n/^\ogn  for  random  problems. 

•  The  algorithm  abjectly  fails  to  generalize,  however,  and,  even  in  the  linearly  separable 
case,  is  inconsistent  with  probability  one  on  the  sample  for  sample  sizes  exceeding 
y/n/^/\ogn]  its  complexity  function  for  perfect  generalization  (for  linearly  separable 
problems)  is  hence  infinite. 


In  lieu  of  formal  proofs  of  these  asymptotic  (as  n  — >  oo)  results,  the  reader  may  wish  to 
examine  the  trends  which  are  apparent  in  Figures  1  and  2.  In  both  cases,  the  consistency 
probability  of  the  algorithm  on  the  sample  is  estimated  by  the  relative  frequency  of  the 
number  of  times  a  solution  was  obtained  for  the  loading  problem  (2)  in  1000  independent 


runs  and  is  shown  plotted  against  normalized  sample  size 


m 


with  dimensionality 


Ti  as  a  parameter.  Observe  the  emergence  of  a  threshold  phenomenon  when  m  is  near 


y/n/y/\ogn  for  large  n  both  for  the  random  and  the  linearly  separable  case.  The  formal 
proofs  devolve  upon  a  consideration  of  the  extreme  tails  of  multivariate  random  walks. 
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m  /  sqrt{n/log(n)) 

Figure  1:  Computer  simulations  of  Harmonic 
Update  on  random  problems. 


m  /  sqrt(n/log{n)) 


Figure  2:  Computer  simulations  of  Harmonic 
Update  on  linearly  separable  training  sets. 


2.3  Directed  Drift 

The  failure  of  Harmonic  Update  to  generalize  efficiently  from  a  linearly  separable  sample 
can  be  traced  directly  to  its  lack  of  consistency  on  large  sample  sizes;  when  m  exceeds 
-v/n/vTogTL,  the  algorithm  does  not  build  up  sufficient  memory  about  the  sample  in  the 
weights.  Two  distinct  approaches  may  be  adopted  to  shore  up  this  paucity  of  memory  at 
the  expense  of  the  rather  parsimonious  space  and  time  complexity  requirements  of  Harmonic 
Update.  In  one  approach,  we  may  choose  to  augment  algorithm  memory  by  increasing  the 
space  complexity  of  the  algorithm,  by  say,  moving  to  an  off-line  procedure;  this  approach  is 
considered  in  the  following  section.  In  an  alternative  approach,  we  may  choose  to  retain  the 
on-line  setting  (with  its  low  attendant  space  complexity  requirements)  and  bolster  algorithm 
memory  of  the  sample  by  repeated  presentations  of  each  example,  thus  increasing  the  time 
complexity  requirements  of  the  algorithm.  This  is  the  approach  considered  in  this  section. 

Recall  that  in  our  discussion  of  Harmonic  Update,  each  example  was  presented  exactly 
once  to  the  algorithm.  The  gentle  reader  may  verify  with  a  quick  calculation  (or  simulation) 
that  allowing  repeated  presentations  of  examples  to  the  algorithm  does  not  improve  its 
generalization  performance.  The  Directed  Drift  family  of  algorithms  that  we  consider  next, 
is  not  quite  so  efficient  in  a  single  pass  of  the  data,  but  eventually,  on  repeated  presentations 
of  the  examples  gradually  builds  up  a  satisfactory  representation  of  the  underlying  concept. 
The  cost  is  time. 

Let  U  be  any  set  of  positive  examples,  and  let  {u[t]}  be  any  training  sequence  in  which 
each  of  the  patterns  in  U  appears  infinitely  often.  Let  {w[t]}  denote  a  binary  learning  se¬ 
quence.  For  each  epoch,  t,  we  denote  by  I[t]  the  subset  of  indices  for  which  the  corresponding 
components  of  w[t]  and  u[t]  are  opposite  in  sign: 

I[t]  =  {i:wi[t]  7^iii[t]  }. 

In  the  simplest  version  of  Directed  Drift,  no  niore  than  a  single  component  of  the  weight 
vector  is  updated  per  epoch. 

Algorithm  D  {Directed  Drift).  Given  any  sequence  of  positive  examples  {  u[t],t  >  1  } 
drawn  from  B^,  the  algorithrh  produces  a  sequence  of  hypotheses  {w[t]  e  B’^jt  >  1  } 
on-line. 

Dl.  [Initialize.]  Set  t  e-  1  and  select  an  arbitrary  initial  hypothesis  w[l]  €  B^. 
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D2.  [Is  the  hypothesis  consistent  on  the  example?]  Set  Y  <r-  {w[t],u[t]). 

D3.  [If  it  ain’t  broke,  don’t  fix  it.]  If  Y  >  0,  set  w[t  -h  1]  <—  w[t]  and  go  to  Step  D5. 

D4,  [Update  hypothesis.]  Else  (if  Y  <  0)  denote  by  J[t]  =  { j  :  'Wj[t]  ^  Uj[t]  }  the  subset 
of  indices  for  which  the  corresponding  components  of  w[t]  and  u[t]  are  opposite  in 
sign,  pick  a  random  index  j[t]  from  the  uniform  distribution  on  J[t],  and  form  the  new 
hypothesis  w[t  +  1]  according  to  the  following  rule: 


Wj[t+  1] 


+Wj[t]  ifj^j[t], 
~Wj(t]  ifj=j[t]. 


D5.  [Increment  time  and  iterate.]  Set  t  t  +  1  and  go  back  to  Step  D2. 


The  intuition  behind  the  algorithm  is  as  follows.  If  a  binary  solution  vector  w®  E 
exists  [i.e.,  the  exemplar  sequence  {u[t]}  is  drawn  from  some  underlying  positive  half-space 
B!J:(w®)]  then  necessarily  we  must  have  (w®,u)  =  ^  ^  for  each  example  n  eU, 

As  there  is  a  contribution  of  -1-1  to  the  sum  if  two  corresponding  components  of  w®  and  u 
have  the  same  sign,  and  ~1  if  the  signs  are  mismatched,  it  follows  that  the  binary  solution 
vector  has  more  component  sign  matches  than  mismatches  with  each  example  in  U. 

Now  the  algorithm  updates  the  current  hypothesis  w[t]  if,  and  only  if,  the  hypothesis 
is  not  consistent  on  the  current  example  u[t]  in  which  case  a  randomly  chosen  mismatched 
component  j[t]  €  J[t]  of  the  offending  hypothesis  is  flipped  (i.e.,  aligned  with  the  sign 
of  the  corresponding  component  of  the  example).  Since,  by  definition,  the  number  |j[t]| 
of  mismatched  components  exceeds  tl/2,  by  the  pigeonhole  principle  there  is  at  least  one 
member  of  J[t]  for  which  the  corresponding  components  of  u[t]  and  w®  have  the  same  sign. 
It  follows  that  there  is  a  positive  probability  that  the  hypothesis  update  is  in  the  direction 
of  the  target  binary  perceptron  w®.  Indeed,  this  suffices  to  show  that  Directed  Drift,  with 
probability  one,  converges  in  finite  time  to  a  hypothesis  consistent  on  any  subset  of  positive 
examples  U  whose  members  appear  infinitely  often  in  the  sequence  {u[t]}. 

We  can  speed  up  convergence  by  considering  a  batch  version  of  Directed  Drift  which 
substitutes  a  democratic  voting  procedure  for  Step  D4. 


Algorithm  B  [Boosting  algorithm).  The  algorithm  receives  as  input  a  hypothesis  w  and 
a  batch  of  m  positive  examples  u\  . . .  ,  which  forms  the  electorate.  The  algorithm 
distributes  an  n-dimensional  ballot  to  each  voter  (example)  and  returns  the  results  of  the 
election  b  =  (bi , . , .  jbn)  where,  for  each  i,  bi  takes  only  integer  values  from  0  to  m  and 
is  the  tally  of  votes  on  the  desirability  of  changing  the  sign  of  the  ith  component  of  the 
hypothesis. 

Bl.  [Distribute  the  ballot.]  For  each  s  =  1,  . . .  ,  m,  form  the  Boolean  vector  (l® , . . .  ,1^) 
where,  for  i  =  1 ,  , . .  ,  n, 


1  ifuf^Wi, 

0  ifu®=wi. 


(5) 


B2.  [Hold  an  election.]  For  each  i  =  1 ,  . , .  ,  n,  tally  the  vote 

m 

bi  =  X^i- 

S  =  1 


Return  the  results  of  the  election  b  =  (bi , . . .  ,bn)- 
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If  a  batch  of  m  =  mb  examples  can  be  called  by  the  algorithm  at  need,  we  may  replace 
Step  D4  by  the  following  more  efficient  Step  D4'  yielding  a  batch  incarnation  of  the  plain 
vanilla  Directed  Drift  algorithm: 

D4'.  [Update  hypothesis.)  Else  (if  Y  <  0)  call  m—  1  additional  examples  and  pass  the 
hypothesis  w  and  the  batch  of  m  positive  examples  ...  ,  to  Algorithm  B 

as  parameters.  Prom  the  results  of  the  election,  b  =  (bi , . . .  jb^),  returned  by  Algo¬ 
rithm  B,  pick  an  arbitrary  index  j  in  the  set  {k  :  >  bt,  1  <  i  <  n}.  Set  Wj  i - Wj 

and  leave  the  other  components  of  w  unchanged.  Set  t  t  -h  m  —  1 . 

The  following  is  a  capsule  summary  of  the  main  features  of  the  algorithm.  Formal 
proofs  and  technical  details  may  be  found  in  the  papers  listed  at  the  end  of  the  report. 

•  If  W  C  B!J:(w®)  is  any  subset  of  vertices  whose  members  appear  infinitely  often  in  a 
sample  {u[t]}  of  positive  examples  of  a  binary  perceptron  w®,  then,  with  probability 
one  for  all  batch  sizes.  Directed  Drift  converges  in  finite  time  to  a  hypothesis  which  is 
consistent  on  U.  The  proof  is  based  on  the  equilibrium  probability  distribution  of  the 
states  of  the  finite  Markov  chain  which  represents  the  system. 

•  For  random  problems.  Directed  Drift  (for  all  batch  sizes)  has  a  maximal  capacity 
function  linear  in  n. 

•  For  linearly  separable  problems,  there  exists  an  absolute  positive  constant  cXc  = 

.  1 .44797 . . .  such  that  the  sequence  oCcU  is  an  (upper  upper  bound  for  the)  complexity 
function  of  Directed  Drift  (for  all  batch  sizes). 

•  For  linearly  separable  problems,  the  on-line  version  of  Directed  Drift,  Algorithm  D 
with  a  batch  size  of  1 ,  has  an  exponential  expected  mistake  bound  bounded  below  by 
oce^^-l-O(n)  where  oc  =  1.771866547. . .  and  |3  =  0.139232271  ...  are  absolute  positive 
constants. 

•  The  batch  version  of  Directed  Drift  generalizes  perfectly  with  a  linear  expected  mistake 
bound  0(n)  for  a  quadratic-log  batch  sample  complexity  mb  =  O(nlogn). 

Again,  in  lieu  of  the  formal  proofs  which  can  be  found  in  the  papers  listed  at  the  end  of  the 
report,  the  reader  may  wish  to  examine  the  suggestive  trends  in  Figures  3  and  4.  The  figures 


Batch  size  /  n 


Figure  3:  Computer  simulations  of  the  batch 
version  of  Directed  Drift. 


Figure  4:  Numerical  simulations  suggest  the 
necessary  and  sufficient  batch  size. 


illustrate  the  savings  in  time  that  relatively  small  batch  sizes  can  yield.  For  each  n,  average 
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mistake  bounds  until  the  algorithm  pinpoints  w®  are  obtained  from  1000  independent  runs, 
and  they  are  plotted  against  batch  sizes.  Both  the  mistake  bounds  and  the  batch  sizes  are 
normalized  by  the  dimensionality  n  before  plotting.  For  the  values  of  n  employed,  all  the 
(normalized)  mistake  bounds  saturate  below  1.0  for  larger  batch  sizes,  bespeaking  a  high 
probability  of  positive  drift.  On  the  other  hand,  as  batch  sizes  decrease,  the  exponential 
convergence  behavior  of  the  algorithm  becomes  apparent,  a  consequence  of  the  diminishing 
success  probability.  The  question  of  interest  is  clearly  to  determine  the  tradeoff  between 
batch  size  and  the  expected  convergence  time.  In  particular,  how  large  must  the  batch  size 
rat,  be  to  ensure  that  a  linear  expected  mistake  bound  is  achieved?  It  transpires  that  a 
batch  size  rub  =  O(nlogn)  is  sufficient  to  ensure  a  uniform  positive  drift  probability  of  at 
least  ^  +  e  (and  hence  guarantee  a  linear  expected  mistake  bound  57  +  0(1)).  Numerical 
simulations  suggest  that  a  batch  size  of  0(nlogn)  is  in  fact  both  necessary  and  sufficient; 
see  Figure  4.  The  new  plot  results  from  a  slight  transformation  of  Figure  3;  batch  sizes  are 
normalized  by  nlogn  prior  to  plotting,  and  p-axis  is  zoomed  in  to  show  mistake  bounds 
up  to  3n  only.  Note  that  data  curves  of  the  five  n  values  intersect  near  (0.24, 1.2),  where 
a  threshold  seems  to  emerge — if  the  batch  size  used  exceeds  0.24nlogn,  the  corresponding 
mistake  bound  appears  to  become  arbitrarily  close  to  O.Sn  as  n  — >  00;  on  the  other  hand,  if 
the  batch  size  employed  falls  below  the  critical  value,  the  associated  mistake  bound  seems 
to  explode  exponentially  without  bound  as  n  increases.  It  follows  that  with  a  quadratic-log 
sample  complexity  ©(n^logn),  batch  version  Directed  Drift  generalizes  perfectly  with  a 
linear  expected  mistake  bound  O(rL). 

2.4  Majority  Rule 

If  off-line  procedures  are  permitted,  substantial  gains  in  capacity  and  learnability  can  be 
made  as  we  illustrate  in  the  following  low-complexity  off-line  algorithm  we  dub  Majority 
Rule.  Let  U  =  as  usual  denote  the  m-set  of  positive  examples  for  loading 

problem  (2).  The  algorithm  prescribes  weights  as  follows: 

For  i  =  1 ,  . . .  yUj  let 


Ut  =:{u€ZY:Ui  =  +1}, 
Ur  ={ueZY:ui  =  «-1 }. 


Set 


if|U+|>lUr|, 

if|U+l<|Ur|. 


In  other  words,  Wi  =  +1  if  patterns  whose  ith  component  is  +1  are  in  the  majority,  and 
=  —1  otherwise.  An  examination  of  the  Loading  Problem  (2)  readily  yields  the  intuitive 
basis  of  the  algorithm.  We  require  each  of  the  sums  to  be  positive  and 

this  is  best  brought  about  if  each  weight  Wi  is  chosen  so  as  to  make  it  most  likely  that  the 
summand  WiUf  is  positive,  i.e.,  if  Wi  and  uf  are  likely  to  agree  in  sign.  Note  that  the 
algorithm  is  homogeneous  and  operates  off-line. 

The  following  is  a  capsule  summary  of  the  main  features  of  the  algorithm.  Formal  proofs 
and  technical  details  may  again  be  found  in  the  papers  listed  at  the  end  of  the  report. 

•  The  memory  requirement  (space  complexity)  of  Majority  Rule  is  mn+n  bits  (mn  bits 
for  the  training  set,  n  bits  for  the  hypothesis  weight  vector).  For  each  component  of 
the  binary  neuron,  the  algorithm  performs  at  most  m—  1  additions  of  bits  and  one  sign 
operation.  Thus  the  (serial)  time  complexity  is  mn,  also  the  minimum  possible.  As  in 
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the  case  of  Harmonic  Update,  locality  and  homogeneity  of  Majority  Rule  allow  us  to 
speed  up  the  hypothesis  generation  process  by  running  n  instances  of  the  algorithm, 
one  for  each  component,  in  parallel. 

•  The  sequence  n/nlogn  is  a  capacity  function  of  the  Majority  Rule  algorithm  for 
random  problems.  Figure  5  provides  empirical  support  of  the  asymptotic  theoretical 
result.  In  the  figure,  the  relative  frequency  of  the  number  of  times  a  solution  was 


Figure  5:  Computer  simulations  of  Majority  Rule  on  random  problems.  The  consistency 
probability  PMR(Ti'>'ra)  is  plotted  versus  sample  size  m  (normalized  by  n/7rlogn)  with  n  as 
a  parameter. 

obtained  for  the  loading  problem  (2)  in  1000  independent  runs  is  shown  plotted  against 
normalized  sample  size  dimensionality  n  as  a  parameter.  Observe  the 

emergence  of  a  threshold  phenomenon  when  m  is  near  n/7tlogn  for  large  n. 

•  For  linearly  separable  problems.  Majority  Rule  is  not  consistent  on  the  examples.  That 
is,  if  Majority  Rule  is  applied  to  the  m-sample  set  W,  the  perceptron  w  generated  does 
not  necessarily  satisfy  (2),  In  other  words,  the  algorithm  does  not  always  load  the 
sample  set.®  Nevertheless,  it  exhibits  superior  generalization  performance  with  low 
sample  complexity.  More  specifically,  we  show  that  the  probability  of  error  incurred 
in  estimating  the  target  perceptron  w®  by  the  hypothesis  w  can  be  made  as  small  as 


^  While  learning  theory  predicts  that  any  consistent  algorithm  will  generalize  given  a  set  of  examples  in 
excess  of  the  VC-dimension,  practical  time  constraints  on  the  algorithms  force  many  heuristics  in  use  to 
be  inconsistent.  Examples  of  inconsistent,  nonetheless  popular,  learning  algorithms  abound,  and  include 
such  generic  classes  as  gradient  descent  algorithms.  For  the  present  problem  too,  the  NP-completeness  of 
binary  integer  programming  makes  it  unlikely  that  there  is  any  polynomial  time  consistent  algorithm  which 
is  guaranteed  to  work  on  all  instances  of  the  problem. 
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we  choose  with  only  a  linear  number  of  examples  m  =  cn.  Furthermore,  the  error 
probability  decreases  exponentially  in  m/n. 

•  As  n  approaches  infinity,  the  Majority  Rule  algorithm  exhibits  an  abrupt  “phase 
transition”  to  zero-error,  perfect  generalization  at  when  the  number  of  examples  (the 
sample  complexity)  reaches  the  critical  number  m  =  Ttnlogn.  That  is,  it  exactly 
learns  the  underlying  majority  function  (i.e.,  the  target  perceptron  w*  £  with  high 
confidence  when  m  exceeds  rm  log  n.  On  the  other  hand,  when  the  number  of  examples 
is  fewer  than  Ttnlogn,  zero-error  generalization  is  not  achievable.  Thus  asymptotically, 
the  algorithm  needs  sample  complexities  for  perfect  learning  only  slightly  in  excess 
(within  a  logarithmic  factor)  of  the  minimum  needed  to  identify  the  function. 

•  Because  the  examples  are  drawn  uniformly  firom  B^,  the  perfect  generalization  re¬ 
sult  has  the  interesting  implication  that  almost  all  instances  of  the  binary  integer 
programming  problem  are  tractable  when  m  exceeds  Ttn  log  n. 

In  Figures  6  and  7,  compelling  empirical  evidence  is  presented  in  support  of  the  strange 
peregrination  of  the  algorithm  towards  perfect  generalization.  In  both  figures,  an  empirical 
estimate  of  the  probability  PMR(Ti,m)  that  the  algorithm  is  consistent  on  a  random,  lin¬ 
early  separable  m-sample  is  plotted  versus  (normalized)  sample  size  (,^/„7ogn  Figure  6 
and  in  Figure  7).  In  brief,  the  algorithm  is  asymptotically  consistent  when  the 


m  /  (it  n  log{n)) 


Figure  6:  Simulations  of  Majority  Rule  on  Figure  7:  Simulations  of  Majority  Rule  on 
linearly  separable  problems:  small  sample  linearly  separable  problems:  large  sample 
regime.  regime. 

sample  size  is  less  than  the  first  threshold  of  fails  abruptly,  i.e.,  is  not  consistent 

on  the  sample,  when  the  sample  size  exceeds  the  first  threshold,  but  recovers  triumphantly 
from  the  dead  when  the  sample  size  exceeds  the  second  threshold  of  rm  log  n.  We  dub  this 
idiosyncratic  behavior  asymptotic  consistency.  At  the  second  threshold — the  sample  com- 
plexity  701  log  n  at  which  consistency  secures  so  remarkably — exceeds  the  1 .45n  information- 
theoretic  estimate  of  Lyuu  and  Rivin  (albeit  only  by  a  logarithmic  amount)  of  the  sample 
complexity  at  which  the  target  perceptron  is  identified  uniquely.  Consequently,  this  implies 
that  Majority  Rule  will  in  fact  generate  the  target  perceptron,  i.e.,  generalize  perfectly,  when 
m  exceeds  TOilogn,  The  proofs  of  these  results  devolve  upon  a  Poisson  clumping  argument 
in  the  very  large  deviation  tails  of  multivariate  random  walks. 
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3  Optimal  Stopping  and  Machine  Complexity  in  Learn¬ 
ing 

See  /18y-/28y. 

The  primary  goal  of  learning  in  a  network  setting  is  to  find  a  network  that  gives  valid 
generalization.  In  achieving  this  goal,  the  network  designer  is  typically  faced  with  the  prob¬ 
lem  of  appropriate  network  size  selection  so  as  to  optimize  the  central  trade-off  between 
training  error  and  network  complexity.  This  issue  has  drawn  much  research  effort  in  re¬ 
cent  years  and  manifold  principles,  theories,  and  intuitions,  including  Occam’s  razor,  and 
statistical  model  selection  criteria  such  as  Akaike’s  information  criterion  (AIC),  Rissanen’s 
minimum  description  length  principle  (MDL),  and  many  others  such  as  Barron’s  complexity 
regularization  method  have  been  developed.  These  principles  all  qualitatively  support  the 
following  suggestive  prescription  from  Vapnik-Cervonenkis  (VC)  theory:  between  two  ma¬ 
chines  which  have  the  same  empirical  error,  the  one  with  smaller  VC-dimension  generalizes 
better.  These  methods,  however,  are  typically  valid  only  at  the  global  minimum  of  the 
empirical  error  function  (for  example,  the  likelihood  function  for  AIC),  and  furthermore,  do 
not  necessarily  lead  to  optimal  (or  even  nearly  optimal)  generalization  performance.  Two 
central  issues  remain  improperly  understood:  (1)  How  is  the  generalization  error  affected 
by  network  complexity  when  there  are  a  finite  number  of  training  examples?  (2)  How  does 
the  generalization  performance  evolve  in  time  as  training  progresses?  We  have  addressed 
these  issues  (and  in  some  cases  found  comprehensive  solutions)  in  the  context  of  learning  a 
target  machine  from  a  finite  collection  of  random  examples  generated  by  it.  We  summarize 
our  main  results  below. 

Generalization  performance  is  not  overtly  affected  by  network  complexity  (size)  when 
the  sample  size  is  infinite.  Indeed,  at  least  nominally,  in  the  limit  of  large  sample  sizes,  valid 
generalization  requires  that  learning  be  consistent,  or  at  least  approach  the  best  approxi¬ 
mation  to  the  target  function  in  the  model  class.  However,  network  complexity  manifests 
itself  explicitly  in  the  generalization  performance  when  the  number  of  available  examples 
is  finite.  Given  a  fixed,  finite  sample,  how  then  should  one  go  about  finding  a  network 
of  proper  complexity  for  which  the  generalization  error  is  minimized?  This  is  the  crux  of 
the  first  issue  raised  above.  A  satisfactory  solution  to  this  problem  requires  the  explicit 
characterization  of  the  effect  of  the  learning  algorithm  on  the  “effective”  complexity  of  the 
network. 

The  second  issue  relates  to  the  time  dynamics  of  the  learning  process.  On  the  empirical 
front,  it  has  often  been  observed  that  during  network  training  by  gradient  descent  there  exists 
a  critical  region  in  time  at  which  the  trained  network  generalizes  best,  following  which  period 
the  generalization  error  actually  increases.  (This  latter  phenomenon  is  ofttimes  referred  to 
as  “over-training”  or  “fitting  the  noise.”)  Furthermore,  in  this  critical  region,  the  size  of  the 
network  appears  to  play  little  role  in  the  (best)  generalization  performance  (at  least,  as  long 
as  the  network  is  large  enough  to  load  the  examples) .  These  observations  seem  to  contradict 
the  learning-theoretic  interpretation  of  Occam’s  razor!  Are  these  numerical  results  just  a 
fluke?  If  not,  when  should  one  stop  learning,  and  how  should  one  define  the  complexity  of  a 
network  and  go  about  adjusting  it  so  that  optimal  generalization  performance  is  achieved? 

Although  relevant  learning  processes  have  been  treated  by  numerous  authors,  the  formal 
theoretical  study  of  these  problems  has  been  abeyant. 

We  attack  these  issues  in  the  framework  of  learning  in  a  general  class  of  machines  which 
return  a  (variable)  linear  form  of  a  (fixed)  set  of  nonlinear  transformations  of  points  in  an 
input  space.  The  study  of  this  simple  machine  class  contains  the  key  elements  for  the  study 
of  the  generalization  dynamics  of  general  nonlinear  learning  machines  without  too  much  of 
the  focus  obscured  by  technical  detail.  The  machines  under  consideration  are  also  interesting 
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in  their  own  right,  since  many  practical  problems  can  be  modeled  by  this  machine  class  and 
the  important  special  case  of  a  class  of  depth-two  neural  networks  with  a  fixed  nonlinear 
front  end  and  a  variable  linear  output  element  reduces  to  this  model;  these  structures  are 
also  related  to  the  O-machines  considered  by  T.  M.  Cover.  In  [22],  [23],  and  [24]  we  report 
extensions  of  the  theory  to  general  parametric  nonlinear  learning  machines  (such  as  general 
feedforward  neural  networks)  where  the  basis  functions  are  not  fixed.  It  transpires  that  the 
flavor  of  the  main  results  derived  in  [27],  [28]  carry  over  to  the  general  nonlinear  case  though 
the  generalization  behavior  for  the  nonlinear  case  turns  out  to  be  much  richer  than  for  the 
linear  case,  largely  because  of  the  increased  complexity  of  the  error  function. 

The  learning-theoretic  issues  raised  above  boil  down  to  a  single  question:  For  a  given 
learning  algorithm,  how  does  the  generalization  error  depend  on  the  network  parameters 
and  the  number  of  examples  at  each  epoch  of  the  learning  process?  In  a  sense,  the  main 
focus  of  [27],  [28]  is  in  providing  a  comprehensive  answer  to  this  question.  In  this  paper  we 
develop  a  rigorous  characterization  of  the  time-dynamics  of  generalization  (for  the  class  of 
machines  considered)  when  a  finite  sample  of  examples  is  available  and  training  is  carried 
out  by  minimization  of  the  empirical  error  via  gradient  descent;  the  paper  is  devoted  to 
a  development  of  the  main  result  and  to  an  exploration  of  its  ramifications  to  network 
complexity,  optimal  stopping,  network  size  selection,  and  regularization.  In  the  following 
we  briefly  sketch  the  principal  results. 

The  empirical  minimum  mean-square  es¬ 
timate  of  the  coefficient  vector,  which  corre¬ 
sponds  to  the  estimate  obtained  in  the  limit 
of  training  over  an  infinity  of  time  steps,  is  un¬ 
biased  and  consistent.  Should  we  then  carry 
training  out  to  its  limit?  Surprisingly,  perhaps, 
the  answer  is  “No.”  This  assertion  follows  as  a 
consequence  of  our  MAIN  THEOREM  which  re¬ 
lates  the  generalization  error  at  each  epoch  of 
learning  to  that  at  the  global  minimum  of  the 
training  error  (in  the  limit  of  training).  As  a 
consequence  of  the  theorem  we  are  able  to  show 
analytically  that  as  training  progresses  in  time 
three  distinct  phases  in  generalization  dynamics 
are  evidenced.  In  the  first  phase,  the  general- 
Figure  8:  The  three  phases  in  the  time  ization  error  decreases  monotonically  (keeping 
evolution  of  generalization.  (Not  drawn  pace  with  a  corresponding  decrease  in  the  train- 
to  scale.)  Best  generalization  obtains  if  ing  error);  this  phase  is  completed  in  O(logTi) 
training  is  optimally  stopped  during  the  time  steps  where  n  is  the  number  of  examples, 
second  phase.  The  behavior  grows  more  interesting  in  the  sec¬ 

ond  phase  where  the  generalization  error  ex¬ 
hibits  complex  dynamics  and  an  optimal  stopping  time  topt  is  evidenced  at  which  the  small¬ 
est  generalization  error  obtains;  the  second  phase  is  also  ephemeral  and  takes  only  O(logu) 
time  steps.  Finally,  in  the  third  phase,  the  generalization  error  increases  monotonically  to  a 
limiting  value  of  error;  this  phase  takes  the  rest  of  time.  The  three  phases  in  generalization 
are  indicated  schematically  in  Figure  8.  Thus,  best  generalization  occurs  not  at  the  limit  of 
training  when  the  global  minimum  of  the  training  error  is  achieved,  but  rather  after  a  finite 
number  of  steps  of  the  order  of  the  logarithm  of  the  sample  size.  Our  main  theorem  also 
provides  explicit  bounds  on  the  optimal  stopping  time  and  on  the  expected  improvement  in 
generalization  error  when  training  is  optimally  stopped. 

One  of  the  key  concepts  that  emerges  from  our  analysis  is  the  formal  notion  of  the  effec- 
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five  size  of  a  network.  This  is  a  time-varying,  algorithm-dependent  quantity  which,  in  the 
limit  of  training  over  an  infinity  of  time  steps,  coincides  with  the  VC-dimension  of  the  ma¬ 
chine.  Our  main  theorem  allows  us  to  characterize  the  relation  between  this  time-dependent 
notion  of  machine  size  and  generalization  error.  As  is  well  known,  a  salient  characteristic 
of  neural  networks  is  that  they  often  involve  a  very  large  number  of  adjustable  parameters 
as  compared  to  traditional  statistical  models  (such  as  classification  and  regression  models) 
with  a  resulting  large  VC-dimension.  For  a  given  (small)  sample  of  fixed  size,  how  then  does 
one  explain  empirical  claims  reporting  valid  generalization?  Our  results  shed  light  on  this 
puzzle:  stopping  learning  at  the  optimal  time  results  in  a  network  with  small  complexity 
in  the  sense  that  its  effective  size  at  that  time  is  typically  substantially  smaller  than  its 
effective  size  in  the  limit  of  training  (the  VC-dimension).  More  generally j  we  show  that  the 
generalization  error  of  the  machine  during  the  training  process  is  determined  at  each  train¬ 
ing  epoch  by  the  effective  size  of  the  machine  at  that  epoch  rather  than  its  VC- dimension. 
Our  analysis  provides  a  formal  framework  within  which  optimal  stopping  can  be  viewed  as 
dynamically  fitting  machine  complexity  to  the  sample  wherein  best  generalization  obtains 
when  effective  machine  size  best  fits  the  sample  size.  Thus  we  rescue  the  prevailing  intuition 
(Occam’s  razor)  from  its  impending  dilemma. 

The  study  of  generalization  dynamics  leads  naturally  to  a  practical  new  network  size  se¬ 
lection  criterion.  The  new  criterion  allows  us  to  simultaneously  obtain  estimates  of  optimal 
network  size  and  optimal  stopping  time  solely  from  a  consideration  of  the  given  examples. 
We  show  that  the  new  criterion  has  an  optimal  character  and  can  be  viewed  as  a  generaliza¬ 
tion  of  Akaike’s  well-known  information  criterion  to  cover  not  just  network  complexity  (the 
effective  machine  size  in  the  limit  of  training)  but  the  time  evolution  of  the  learning  process 
as  well.  The  analysis  in  [22]  and  [24]  also  allows  of  a  comparison  between  various  criteria 
and,  in  particular,  leads  to  a  better  understanding  of  the  relative  merits  of  approaches  based 
on  the  minimum  description  length  principle  vis  a  vis  Akaike’s  information  principle. 

Regularization  has  been  touted  as  a  mechanism  by  which  the  performance  of  the  gradi¬ 
ent  descent  algorithm  can  be  improved.  Our  analysis  shows  that  the  two  methodologies  of 
regularization  and  optimal  stopping  are  distinct  in  spite  of  sharing  superficial  similarities. 
In  particular,  for  the  general  class  of  machines  considered  in  [27],  [2S],  we  find  that  regu¬ 
larization  frequently  results  in  inconsistent  learning  with  poorer  generalization  while  optimal 
stopping  always  leads  to  consistent  learning  with  improved  generalization. 

Extensions  of  these  results  to  general  parametric  nonlinear  machines  and  neural  net¬ 
works  and  to  other  learning  algorithms  are  detailed  in  [22],  [23],  and  [24].  It  transpires  that 
the  main  results  derived  here  carry  over  to  the  general  nonlinear  case  even  when  the  objec¬ 
tive  function  is  not  restricted  to  the  squared-error  loss  function.  This  is  true  for  all  results 
except  those  on  the  first  phase  of  learning  where  different  methods  from  those  employed 
here  must  be  used  to  study  the  problem.  The  generalization  behavior  for  the  nonlinear 
case  turns  out  to  be  much  richer  than  for  the  linear  case,  largely  because  of  the  increased 
complexity  of  the  error  function. 

For  practical  applications  of  neural  networks,  our  results  demonstrate  that  training  a 
network  to  its  limit  is  not  desirable.  Thus,  the  inability  of  backpropagation  networks  to  find 
the  global  minimum  of  the  empirical  error  is  not  necessarily  a  drawback;  it  may  precisely  be 
the  fact  that  global  minima  of  the  empirical  error  were  not  attained  that  has  contributed 
to  successful  applications  of  neural  networks  in  the  past! 

While  the  problem  of  inferring  a  rule  from  the  observed  data  has  been  studied  for  a 
long  time  in  learning  theory  as  well  as  in  other  contexts  such  as  in  linear  and  nonlinear 
regression,  the  study  of  the  problem  as  a  dynamical  process  seems  to  open  up  a  new  avenue 
for  looking  at  the  problem.  Many  problems  are  still  open,  however,  and  we  are  currently  at¬ 
tempting  to  extend  our  approaches  to:  systematically  characterize  and  use  side- information 
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in  the  learning  problem;  determine  the  appropriate  node  functions  for  a  given  problem;  and 
characterize  the  effect  of  various  nested  architectures  and  learning  algorithms. 


4  Nonparametric  Learning 

See  [9j,  [12]-[15j. 

A  characteristic  of  nonparametric  algorithms,  such  as  the  k-nearest  neighbor  classifier,  ker¬ 
nel  density  estimation,  and  neural  networks  that  acquire  new  hidden  units  as  the  training 
set  is  increased,  is  that  they  are  able  to  learn  solutions  with  nearly  optimal  accuracy  using 
minimal  prior  information.  Consequently,  T.  M.  Cover  has  conjectured  that  the  classi¬ 
fication  accuracy  of  the  k-nearest  neighbor  algorithm  estimates  the  accuracy  of  the  best 
nonparametric  classifier  that  uses  the  same  set  of  labeled  patterns.  This  conjecture  rests 
on  the  proven  asymptotic  consistency  of  nearest  neighbor  algorithms,  and  the  difficulty  of 
incorporating  side  information  into  algorithms  of  such  generality.  That  difficult  pattern 
recognition  problems  usually  do  not  exhibit  useful  side  information,  also  suggests  that  the 
sample  complexity  of  the  k-nearest  neighbor  algorithm  may  estimate,  in  a  loose  sense,  the 
intrinsic  complexity  of  such  problems.  The  k-nearest  neighbor  algorithm,  in  particular,  is  a 
close  relative  of  unsupervised  neural  network  learning  algorithms  such  as  Kohonen’s  learning 
vector  quantization  algorithm  and  feature  maps.  In  [13]  and  [14]  we  provide  perhaps  the 
first  rigorous  estimates  of  the  performance  of  the  k-nearest  neighbor  algorithm  on  a  finite 
sample  of  examples  and  an  identification  of  optimal  choice  of  metric.  These  results  allow 
one  to  delineate  in  a  fairly  precise  fashion  how  much  information  each  example  carries  and 
yield  qualitative  information  on  the  likely  efficacy  of  growth  algorithms  in  neural  networks. 

In  the  following  we  summarize  the  main  results  of  [14].  In  its  original  form,  the  k- 
nearest-neighbor  classifier  is  perhaps  the  simplest  pattern  classification  algorithms  yet  de¬ 
vised.  Given  a  classification  problem,  in  which  a  pattern,  represented  as  an  n-dimensional 
feature  vector,  is  to  be  assigned  to  one  of  several  pattern  classes  (e.g.,  states  of  nature),  the 
k-nearest-neighbor  algorithm  requires  three  ingredients:  (i)  Si  finite  reference  sample  of  m 
correctly  classified  feature  vectors  from  the  given  problem,  {ii)  an  integer  k  (with  k  <  m), 
and  (Hi)  a  metric,  or  pattern  similarity  function.  Given  an  input  feature  vector,  the  k 
feature  vectors  from  the  reference  sample  that  are  closest  to  the  input  vector  are  identi¬ 
fied  using  the  metric.  The  input  feature  vector  is  then  assigned  to  the  class  that  appears 
most  frequently  amongst  the  k  nearest  neighbors.  (If  no  single  class  appears  with  greatest 
frequency,  then  an  auxiliary  procedure  can  be  invoked  to  handle  ties.) 

Despite  its  simplicity,  this  nonparametric  algorithm  is  asymptotically  consistent  with  a 
Bayes  classifier,  and,  for  a  variety  of  practical  applications,  the  accuracy  of  the  algorithm  is 
competitive  with  leading  pattern  classifiers.  Its  favorable  performance  in  the  infinite-sample 
limit  (m  — >  oo)  was  revealed  in  the  seminal  analysis  of  Cover  and  Hart  in  1967.  Their 
findings  were  later  strengthened  and  generalized  in  a  variety  of  studies  which  demonstrate 
that  if  the  reference  sample  and  input  patterns  are  independently  selected  at  random  by 
arbitrary,  stationary  distributions;  then  the  statistical  risk  of  the  k-nearest  neighbor  classifier 
(e.g.,  the  probability  that  a  random  input  vector  is  misclassified)  tends  to  a  value  that  is  close 
to  the  Bayes  risk  (viz.,  the  optimal  misclassification  probability).  Specifically,  as  m  — ^  oo: 

1.  If  k  =  1 ,  then  the  statistical  risk  tends  to  a  value  that  does  not  exceed  twice  the  Bayes 
risk;  and 

2.  If  k  — >  oo  such  that  k/m  — >  0,  then  the  statistical  risk  tends  to  the  Bayes  risk. 

In  [14],  we  examine  how  the  finite-sample  risk  of  the  k-nearest  neighbor  algorithm 
depends  upon  the  selection  of  its  metric.  We  specifically  consider  a  weighted  Lp  metric  of 
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the  form  d(x,y)  =  ||A(x  — y)||p,  where  x,  y  €  A  is  a  constant,  nonsingular,  linear 
transformation,  and  |lx||p  represents  the  Lp  norm  of  x.  This  classifier  is  then  applied 
to  a  smooth  family  of  classification  problems  Jn  described  by  probability  densities  with 
uniformly  bounded  partial  derivatives  up  through  order  N  +  1 . 

Under  certain  ancillary  conditions  on  the  class  Tn,  we  obtain  an  exact  integral  ex¬ 
pression  for  the  finite-sample  probability  of  error  of  the  k-nearest  neighbor  classifier  by 
evaluating  the  error  probability  asymptotically  using  a  multidimensional  generalization  of 
Laplace’s  method.  This  analysis  yields  a  truncated  asymptotic  representation  for  the  finite- 
sample  probability  of  error  in  the  form 

N 

j=2 

Here,  Pqo  is  the  infinite-sample  risk  probability  of  error  derived  by  Cover  and  Hart  (1967), 
and  is  independent  of  the  choice  of  metric.  The  expansion  coefficients  C2,  C3,  . . .  ,  however, 
depend  upon  k,  the  metric  parameters  p  and  A,  and  the  chosen  procedure  for  handling 
ties,  in  addition  to  the  probability  distributions  that  describe  the  classification  problem 
under  consideration.  This  result  applies  to  pattern  recognition  problems  from  Jn  with  a 
finite  number  of  pattern  classes  and  a  deterministic  tie-breaking  algorithm.  (Extensions 
to  classifiers  that  use  a  random  procedure  (e.g.,  tossing  a  fair  coin)  to  resolve  ties  can 
be  obtained  by  averaging  Eqn.  (6)  over  an  appropriately  weighted  ensemble  of  different 
deterministic  procedures.)  By  analyzing  the  leading  coefficients  in  (6),  we  establish  the 
following: 

For  every  C-class  problem  within  J2,  the  weighted  Euclidean  metric  (Le.,  p  =  2 j 
minimizes  the  Gnite-sample  probability  of  error,  with  respect  to  all  other  Lp 
metrics  that  use  the  same  weight  matrix  A,  if  the  size  of  the  reference  sample  is 
sufficiently  large. 

The  asymptotic  optimality  of  the  Euclidean  metric  is  shown  in  detail  for  a  two-class 
problem  with  odd  values  of  k.  Under  these  conditions  the  evaluation  of  the  expansion  coef¬ 
ficients  in  (6)  is  simplified  by  the  practical  elimination  of  ties.  For  this  case  we  have  derived 
explicit  formulas  for  the  leading  coefficients  through  C5;  coefficients  for  other  cases  can  be 
obtained  similarly.  Numerical  simulations  of  k-nearest  neighbor  classifiers  also  confirm  that 
the  L2  metric  is  more  accurate  than  the  and  Lqo  metric  for  normally  distributed  two-class 
problems  in  12  and  25  dimensional  feature  spaces  for  practically  attainable  sample  sizes. 

The  asymptotically  optimal  linear  transformation  A  within  the  metric  depends  upon 
the  underlying  distributions  that  describe  the  given  pattern  recognition  problem.  Given 
the  analytic  forms  of  these  distribution,  this  transformation  can  be  obtained,  in  principle, 
as  the  solution  of  a  constrained  optimization  problem  using  the  Euler-Lagrange  multiplier 
theorem. 

To  our  knowledge,  this  is  the  first  study  to  rigorously  evaluate  the  finite-sample  risk  of 
the  k-nearest  neighbor  classifier  under  different  Lp  metrics.  In  prior  work,  Pukunaga  and 
Flick  (1984)  have  nonrigorously  exainined  the  accuracy  of  this  algorithm  under  a  weighted 
Euclidean  metric;  their  heuristic  expression  for  the  optimal  quadratic  weight  matrix,  how¬ 
ever,  differs  somewhat  from  our  estimates. 

In  ongoing  work,  we  are  seeking  to  further  extend  the  range  of  validity  of  these  results  to 
general  nonparametric  situations,  and  in  particular,  learning  algorithms  in  neural  networks 
when  the  network  structure  is  not  held  fixed,  as  is  the  case  when  lack  of  side-information 
about  the  problem  constrains  an  effectively  nonparametric  search  through  netwprk  space. 
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These  issues  are  also  linked  to  those  discussed  earlier  regarding  net-size  selection  and  opti¬ 
mum  generalization  for  given  data  sets. 


5  Learning  from  a  Mixture  of  Labeled  and  Unlabeled 
Examples 

See  [lOj,  [nj. 

The  classical  problem  of  learning  a  classification  rule  can  be  stated  as  follows:  patterns 
from  classes  “1”  and  “2”  (or  “states  of  nature”)  appear  with  probabilities  =  p  and 
P2  =  1  — p,  respectively;  the  pattern  classes  are  represented  by  feature  vectors  x  in  a  common 
N-dimensional  Euclidean  space  ,  the  patterns  of  class  “i”  distributed  according  to  the 
class-conditional  probability  density  fi(x)  (i  =  1,  2).  Labeled  pairs  (x,y)  6  R^  x  {1,2]  are 
assumed  generated  according  to  the  following  mechanism:  a  pattern  class  (or  “label”)  y  € 

(1 , 2}  is  first  drawn  randomly  according  to  the  distribution  of  classes  (pi ,  P2};  a  corresponding 
random  feature  vector  x  6  R^  is  then  drawn  according  to  the  cletss-conditional  density  iy . 

In  the  supervised  learning  scenario,  a  labeled  m-sample  ( 1  <  j  <  m)  is  acquired  by 
independent  sampling  from  the  distribution  of  pairs  (x,^).  Using  the  sample,  the  objective 
is  to  construct  a  decision  rule  which  when  presented  with  a  random  pattern  x  (drawn  from 
the  mixture  density  f  =  pifi  +P2f2)  produces  a  label  which  disagrees  with  the  true  class  of 
origin  by  a  probability  Perror  close  to  the  minimal  Pnayes  error  rate.  Formally  this  learning 
problem  can  be  formulated  in  the  framework  of  the  Probably  Approximately  Correct  (PAC) 
learning  model  of  Valiant  (1984)  as  follows:  Given  e  >  0,  8  >  0,  and  a  labeled  sample  of 
size  m  =  m(e,  8),  construct  a  classification  rule  which  with  confidence  in  excess  of  1  —  8  has 
an  error  probability  Perror  =  PBayes{l  +  coe).® 

As  is  well  known  in  the  unsupervised  learning  literature  an  unlabeled  teaching  sample 
(xi  e  R^  :  1  <  i  <  n)  drawn  by  independent  sampling  from  the  mixture  density  f{x), 
can  also  be  used  in  the  process  of  learning  classification.  If  the  mixture  probability  density 
function  f{x  1  0)  =  pifi{x  |  0i)  +P2f2(^  I  62)  is  parametric  (with  unknown  parameters 
0  =  [01 , 62]  and  p)  and  is  identifiable  then  the  unlabeled  sample  (xi)  can  be  used  to  generate 
an  identifiable  density  function  f(x  |  0)  as  an  estimate  of  f(x  |  0).  Identifiability  ensures 
that  the  Bayes  optimal  decision  border  (x  :  pifi  (x  |  0^ )  =  P2f2(^  I  02)1  can  be  deduced 
if  f(x  I  0)  is  known  and  therefore  one  can  construct  an  estimate  of  the  Bayes  border  by 
using  f(x  I  0)  instead  of  f(x  |  0)  (such  plug-in  rules  are  asymptotically  optimal).  Once  the 
decision  border  is  estimated,  a  small  labeled  sample  suffices  to  learn  (with  high  confidence 
and  small  error)  the  appropriate  class  labels  li ,  I2  €  (1,2)  associated  with  the  two  disjoint 
decision  regions  in  R^  generated  by  the  estimate  of  the  Bayes  decision  border. 

Clearly,  unlabeled  examples,  albeit  of  less  worth  than  labeled  examples,  can  still  carry 
information  of  value  to  the  learner.  Furthermore,  in  many  cases  of  practical  interest,  un¬ 
labeled  examples  exist  in  profusion  in  nature  (or  are  inexpensively  acquired)  while  their 
labeled  counterparts  are  few  in  number  (or  are  expensive  to  acquire).  For  example,  in  a 
two-class  tree  recognition  problem  “Douglas  Fir”  and  “Spruce,”  unlabeled  examples  of  these 
trees  may  exist  in  profusion  in  a  forest  but  labeling  them  requires  the  services  of  a  human 
expert  who  charges  by  the  hour.  Or,  in  an  (idealized)  cancer  diagnosis  scenario,  obtaining 
x-rays  of  cells  may  be  relatively  cheap  compared  to  the  cost  of  the  expert  labeling  them, 

A  mixed  sample  learning  system  utilizing  both  labeled  and  unlabeled  examples  may  hence 
be  of  interest  where  one  trades  off  expensive  labeled  examples  for  (very  many)  unlabeled 
examples.  In  such  a  scenario,  the  learner  would  like  to  determine  the  exact  tradeoff  between 


®We  use  Co,  Cl ,  C2,  . . .  to  represent  positive  constants  independent  of  the  error  and  confidence  param¬ 
eters  e  and  6  and  the  dimension  of  the  feature  space  N . 
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labeled  and  unlabeled  examples.  This  question  was  first  posed  by  T.  M.  Cover  in  the  fol¬ 
lowing  succinct  but  evocative  form:  How  many  unlabeled  examples  is  each  labeled  example 
worth?  In  recent  work  Castelli  and  Cover  have  shown  that  if  an  infinity  of  unlabeled  ex¬ 
amples  is  available  then  for  identifiable  classes  with  p,  fi  and  1 2  unknown,  the  probability 
of  error  decreases  exponentially  fast  in  the  number  of  labeled  examples  to  the  Bayes  risk; 
they  have  also  shown  that  if  only  a  finite  number  of  labeled  and  unlabeled  examples  are 
available  but  the  class-conditional  densities  fi  and  iz  are  known  with  the  only  unknown 
being  the  mixing  parameter  p,  then  under  suitable  regularity  conditions  labeled  samples  are 
exponentially  more  valuable  than  unlabeled  samples. 

In  [10]  we  consider  a  parametric  situation  where  the  class-conditional  densities  fi(x  |  0i) 
(i  =  1 ,  2)  are  specified  by  a  parameter  (vector)  0^  €  .  We  assume  that  the  learner  is  pro¬ 

vided  with  a  finite  mixed  sample  of  m  labeled  examples  and  n  unlabeled  examples  together 
with  side-information  specifying  the  parametric  form  of  the  class-conditional  densities  (but 
not  the  densities  themselves);  the  parameter  vector  0  =  [0i  ,02]  as  well  as  the  mixing  pa¬ 
rameter  p  are,  however,  unknown  to  the  learner.  Our  goal  is  to  determine  how  the  error 
rate  depends  on  the  sample  sizes  m  and  n  and  on  the  dimensionality  N.  Focusing  on  the 
Gaussian  mixture  case  for  definiteness,  applications  of  uniform  strong  laws  show  that  each 
labeled  example  has  to  be  replaced  by  of  the  order  of 

Niioei_ 

e'^p^^  logN 

unlabeled  examples  to  achieve  the  same  performance.  A  precise  statement  of  this  result 
may  be  found  in  [10]. 

In  nonparametric  cases,  the  tradeoff  between  labeled  and  unlabeled  examples  is  much 
more  pronounced.  For  instance,  if  kernel  density  estimation  is  used  to  infer  the  mixture 
density  of  an  identifiable  family  of  bimodal  mixture  densities  (including  Gaussian  mixtures), 
each  labeled  example  is  worth  roughly 

C2l3^  log{5+logN) 

N  logNe^^TSTFT 

unlabeled  examples.  A  more  precise  statement  of  the  result  may  be  found  in  [11]. 

In  ongoing  work,  we  are  attempting  to  build  side-information  about  the  problem  into 
the  learning  framework. 


6  Enumeration,  Computation,  Learning 

In  tangential  work,  [8]  provides  a  variety  of  results  enumerating  the  number  of  folds  in 
RNA  secondary  structures;  [12]  applies  the  probably  approximately  correct  learning  model 
of  Valiant  in  conjunction  with  the  uniform  strong  laws  of  Vapnik  and  Cervonenkis  to  provide 
resolutions  of  the  learnability  and  detectability  of  fraud  in  various  situations;  [16]  provides 
an  elementary  proof  using  Pythagoras’  theorem  of  the  universal  approximation  capability  of 
depth-two  sigmoidal  neural  networks;  and  [17]  provides  a  comprehensive  general  framework 
for  storage  capacity  issues  in  recurrent  neural  networks. 


See  [S],  [12],  [16],  [17 
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